Representation apparatus for displaying a graphical representation of an augmented reality

ABSTRACT

A representation apparatus for displaying a graphical representation of an augmented reality includes a capture unit, a first display unit, and a processing unit. The first display unit is at least partially transparent. The capture unit is configured to capture a relative positioning of the first display unit relative to a representation area of a second display unit. The processing unit is configured to determine an observation geometry between the first display unit and the representation area of the second display unit based on the relative positioning, receive a dataset, generate the augmented reality based on the dataset, and provide the graphical representation of the augmented reality via virtual mapping of the augmented reality onto the representation area along the observation geometry. The first display unit displays the graphical representation of the augmented reality in at least partial overlaying with the representation area of the second display unit.

This application claims the benefit of German Patent Application No. DE10 2021 206 565.1, filed on Jun. 24, 2021, which is hereby incorporatedby reference in its entirety.

BACKGROUND

The present embodiments relate to a representation apparatus fordisplaying a graphical representation of an augmented reality, a system,and a method for providing a graphical representation of an augmentedreality.

For the realistic representation of medical information (e.g., medicalimage data from an examination object), representations of an augmentedreality (AR) are increasingly being used. Herein, real objects (e.g.,medical objects and/or an examination object) are often displayedoverlaid with virtual data (e.g., medical image data and/or virtualobjects) and represented in a display. For a realistic representationwith a high degree of immersion, a precise registration between thevirtual data and the real objects is to be provided.

A graphical representation of preoperative and/or intraoperative imagedata of an examination object in augmented reality may be used to assista medical personnel member (e.g., a medical practitioner) duringinterventional and/or surgical procedures. However, it is oftendisadvantageous therein that the image data to be displayed by theapparatus for representing the augmented reality is to be received andprocessed by a providing unit and/or a medical imaging device (e.g., inreal time). This may be limited in a disadvantageous manner in the case,for example, of a simultaneous provision of the image data to aplurality of apparatuses for representing an augmented reality, by anavailable transfer bandwidth. Further, an image quality of the imagedata to be displayed by the apparatus for representing the augmentedreality may thereby be disadvantageously reduced. A delay between therecording of the, for example, intraoperative image data of theexamination object and its display in the augmented reality and/or thereduced image quality may lead to a faulty coordination during theinterventional and/or surgical procedure. As a result, a risk of injuryto the examination object may be increased.

In a medical environment, it is often necessary that three-dimensionally(3D) resolved image data of the examination object may be observedsimultaneously by a plurality of observers among the medical personnel.For this purpose, for example, a 3D monitor in combination withstereoscopic filter goggles that may be worn by the medical personnelmay be used. The 3D monitor may therein provide two individualstereoscopic images that may be captured for a plurality of observersamong the medical personnel by the filter goggles. A disadvantage,however, is the lack of adaptation of the stereoscopic single images tothe, for example, momentary observation points of the plurality ofobservers. As a result and, for example, dependent upon the observationpoints, an unrealistic and/or insufficient depth perception may ariseduring the observation of the three-dimensionally resolved image data onthe 3D monitor.

SUMMARY AND DESCRIPTION

The scope of the present invention is defined solely by the appendedclaims and is not affected to any degree by the statements within thissummary.

The present embodiments may obviate one or more of the drawbacks orlimitations in the related art. For example, an improved adaptation ofan augmented reality is provided.

In a first aspect, a representation apparatus for displaying a graphicalrepresentation of an augmented reality is provided. Therein, therepresentation apparatus has a capture unit, a first display unit, and aprocessing unit. Further, the first display unit is configured to be atleast partially transparent. The capture unit is configured to capture arelative positioning of the first display unit relative to arepresentation area of a second display unit. The representation area ofthe second display unit is therein configured to display graphicalinformation. The processing unit is configured to determine anobservation geometry between the first display unit and therepresentation area of the second display unit based on the relativepositioning. In addition, the processing unit is configured to receive adataset. Further, the processing unit is configured to generate theaugmented reality based on the dataset. The processing unit is alsoconfigured to provide the graphical representation of the augmentedreality via a virtual mapping of the augmented reality onto therepresentation area of the second display unit along the observationgeometry. In addition, the first display unit is configured to displaythe graphical representation of the augmented reality in at leastpartial overlay with the representation area of the second display unit.

The second display unit may include, for example, a screen and/or amonitor and/or a projector and/or a projection area. Therein, therepresentation area of the second display unit may include theprojection area and/or a display layer (e.g., panel) of the screenand/or the monitor. For example, the representation area of the seconddisplay unit may include a spatially limited area (e.g., coherent area)on which the graphical information may be displayed. Therein, therepresentation area of the second display unit may extend, at leastpartially (e.g., completely) flat (e.g., planar or curved). Thegraphical information may include image data (e.g., medical image data)and/or text information (e.g., operating parameters and/or objectparameters of an object (medical object)) and/or metadata regarding themedical image data.

The capture unit may be configured to capture a spatial positioning ofthe first display unit and the second display unit (e.g., therepresentation area of the second display unit). The spatial positioningof the first display unit and the second display unit may each include aspatial position and/or alignment of the first display unit and thesecond display unit in a coordinate system of the capture unit. Thecapture unit may be further configured to capture the relativepositioning of the first display unit in relation to the representationarea of the second display unit based on the captured spatialpositioning of the first display unit and the second display unit. Therelative positioning of the first display unit in relation to therepresentation area of the second display unit may include an item ofinformation regarding a spatial distance between the first display unit(e.g., an origin of the coordinate system of the first display unit) andthe representation area of the second display unit (e.g., a referencepoint, such as a midpoint or corner point of the representation area).Further, the relative positioning may include an item of informationregarding an angle (e.g., a viewing angle) of the first display unit inrelation to the representation area of the second display unit (e.g., inrelation to a normal vector in the reference point of the representationarea).

Alternatively, the capture unit may be configured to capture therelative positioning of the first display unit or the second displayunit (e.g., the representation area of the second display unit) inrelation to the other display unit. Therein, the capture unit may bearranged in a defined manner in relation to first display unit or thesecond display unit (e.g., fastened to the first display unit or thesecond display unit) and/or may be at least partially (e.g., completely)integrated into the first display unit or the second display unit. Bythis, the capture unit may be configured to directly capture therelative positioning of the representation area of the second displayunit in relation to the first display unit. Therein, the relativepositioning of the first display unit in relation to the representationarea of the second display unit may describe a, for example,three-dimensional spatial position and orientation of the representationarea in a coordinate system of the first display unit.

The processing unit may be further configured to determine theobservation geometry between the first display unit (e.g., one or moreobservation points, such as eyes, of a user looking through the firstdisplay unit in one operating state of the representation apparatus) andthe representation area of the second display unit based on the relativepositioning. Therein, the observation geometry may describe an opticalray path between the representation area of the second display unit andthe first display unit (e.g., the one or more observation points of theuser). For example, the observation geometry may describe, in theoperating state of the representation apparatus, the optical ray pathfrom the graphical information displayed on the representation area ofthe second display unit, through the first display unit (e.g., thegraphical representation of the augmented reality), to the one or moreobservation points (e.g., the eyes of the user looking through the firstdisplay unit). By this, the observation geometry may describe theoptical ray path of an overlaid observation of the graphical informationthat may be displayed on the second display unit with the graphicalrepresentation of the augmented reality starting from the one or moreobservation points of the user.

The processing unit may be coupled communicatively to the first displayunit and the capture unit (e.g., wirelessly or cable-bound). Thereception of the dataset may include, for example, a capture and/orreadout of a computer-readable data store and/or a reception from a datastorage unit (e.g., a database). Further, the dataset may be provided bya medical imaging device. For this purpose, the processing unit may havean interface.

The dataset may include image data (e.g., medical image data) and/orimage parameters (e.g., metadata) and/or model data (e.g., a patientand/or organ and/or tissue model) and/or model parameters and/or textdata (e.g., operating parameters) and/or object parameters (e.g.,medical object parameters) and/or patient information (e.g., historydata and/or diagnosis data and/or physiological information and/orprocedural data, such as an item of information for a surgery phase).

The processing unit may be configured to generate the augmented realityhaving at least one virtual object (e.g., a plurality of virtual objectsbased on the dataset). Therein, the augmented reality may include, forexample, a two-dimensional and/or three-dimensional virtual arrangementof the at least one virtual object (e.g., the plurality of virtualobjects in the coordinate system of the first display unit). In oneembodiment, the processing unit may be configured to generate the atleast one virtual object having geometrical and/or anatomic and/ortextual and/or graphical features of the dataset. Further, theprocessing unit may be configured to register the augmented reality(e.g., the at least one virtual object) to the representation area ofthe second display unit.

The processing unit may also be configured to provide the graphicalrepresentation of the augmented reality (e.g., a graphicalrepresentation of the at least one virtual object via a virtual mappingof the augmented reality, such as the at least one virtual object) ontothe representation area of the second display unit along the observationgeometry. Therein, the graphical representation may include, forexample, a two-dimensional and/or three-dimensional representation(e.g., stereoscopic representation) of the augmented reality on thefirst display unit. Therein, the graphical representation of theaugmented reality may be arranged, for the user who looks through thefirst display unit, virtually on the representation area of the seconddisplay unit. The virtual mapping of the augmented reality onto therepresentation area of the second display unit along the observationgeometry may include, for example, a virtual arrangement of the at leastone virtual object on the representation area of the second display unitalong the observation geometry. In addition, the virtual mapping forproviding the graphical representation of the augmented reality may bebased, for example, on a virtual projection mapping and/or ray-tracingand/or a rendering according to a mapping matrix (e.g., a camera matrix)of the at least one virtual object onto the representation area of thesecond display unit along the observation geometry. Further, the virtualmapping of the augmented reality onto the representation area of thesecond display unit may include a virtual illumination and/or a virtualshadow of the at least one virtual object according to the observationgeometry (e.g., momentary observation geometry). Further, the processingunit may be configured to adapt a stereo disparity of the virtualmapping dependent on the observation geometry for a stereoscopic displayof the graphical representation of the augmented reality by the firstdisplay unit. In addition, the graphical representation of the augmentedreality may have context information (e.g., an item of patientinformation and/or a workflow indication) that is virtually arrangedoutside the representation area of the second display unit (e.g.,adjoining the representation area and/or in a plane with therepresentation area).

The provision of the graphical representation of the augmented realitymay include a storage on a computer-readable storage medium (e.g., anon-transitory computer-readable storage medium) and/or a transfer to afirst display unit.

The first display unit may be configured to display the graphicalrepresentation of the augmented reality (e.g., two-dimensionally orthree-dimensionally). For example, the first display unit may beconfigured for stereoscopic display of the graphical representation ofthe augmented reality. The first display unit may be configured torepresent real (e.g., physical) objects (e.g., the representation areaof the second display unit), with the graphical representation of theaugmented reality (e.g., a graphical representation of the at least onevirtual object) at least partially overlaid and in a display. For thispurpose, the first display unit may include, for example, a screenand/or a monitor and/or a projector and/or a projection area. In oneembodiment, the first display unit may be configured as goggles (e.g.,data goggles) and/or a helmet (e.g., a data helmet). Further, the firstdisplay unit may be configured to be portable (e.g., wearable by auser). Therein, the first display unit may follow a movement of the user(e.g., a head movement of the user). Further, the first display unit maybe configured at least partially light-transmitting (e.g., translucentand/or transparent). Further, the first display unit may be configuredto be arranged in a field of view of the user.

The embodiment may enable a display of the graphical representation ofthe augmented reality adapted to the observation geometry (e.g.,momentary observation geometry) of the user. Therein, the graphicalrepresentation of the augmented reality may at least partially overlaythe graphical information that the second display unit displays on itsrepresentation area in an operating state of the representationapparatus. Further, in the operating state of the representationapparatus, the second display unit may display the graphical informationfrom the display of the graphical representation of the augmentedreality uninfluenced (e.g., free from distortion and/or in a higherimage quality) on the representation area of the second display unit.

In a further embodiment of the representation apparatus, the processingunit may be configured, in the case of a changed relative positioning,to determine repeatedly the observation geometry between the firstdisplay unit and the representation area of the second display unit.Further, the processing unit may be configured, with the changedrelative positioning, to provide the graphical representation of theaugmented reality based on the most recently determined observationgeometry.

The capture unit may be configured to capture directly the relativepositioning of the first display unit in relation to the representationarea of the second display repeatedly (e.g., continuously). Further, theprocessing unit may be configured to identify a deviation between a mostrecently captured and a previously captured relative positioning of thefirst display unit in relation to the representation area of the seconddisplay unit. Therein, the identification of the deviation may include acomparison between the most recently and the previously capturedrelative positioning. Further, the processing unit may be configured todetermine repeatedly the observation geometry (e.g., based on the mostrecently captured relative positioning). Further, the processing unitmay be configured to provide the graphical representation of theaugmented reality based on the most recently determined observationgeometry. For example, the processing unit may be configured to providethe graphical representation of the augmented reality via the virtualmapping of the augmented reality onto the representation area of thesecond display unit along the most recently determined observationgeometry.

By this, the graphical representation of the augmented reality may beadapted to the relative positioning that has been changed (e.g., by amovement of the user).

In a further embodiment of the representation apparatus, the captureunit may be arranged, spaced from the first display unit and the seconddisplay unit or at least partially (e.g., completely) integrated intothe first display unit or the second display unit.

In a first variant, the capture unit may be arranged, spaced from thefirst display unit and the second display unit (e.g., positionally fixedin relation to a room in which the first display unit and the seconddisplay unit are arranged in an operating state of the representationapparatus). In one embodiment, the capture unit may be arranged (e.g.,positioned), such that the first display unit and the second displayunit are arranged, in the operating state of the representationapparatus, in a capture region of the capture unit. By this, the captureunit may be configured to capture the spatial positioning of each of thefirst display unit and the second display unit (e.g., simultaneously).

In a second variant, the capture unit may be integrated at leastpartially into the first display unit or at least partially into thesecond display unit. Therein, the capture unit may have a defined (e.g.,positionally fixed) arrangement in relation to the first display unit orthe second display unit (e.g., the representation area of the seconddisplay unit). In one embodiment, the capture unit may be integrated atleast partially into the first display unit or the second display unitsuch that the respective other display unit is arranged, in theoperating state of the representation apparatus, in the capture regionof the capture unit. By this, the capture unit may be configured tocapture directly the relative positioning between the first display unitand the second display unit.

In an arrangement of the capture unit spaced from the first display unit(e.g., in an at least partial integration of the capture unit into thesecond display unit), the capture unit may be configured for capture(e.g., simultaneous capture) of the respective spatial positioning(e.g., the relative positioning of each representation apparatus of aplurality of similar or different representation apparatuses).

In a further embodiment of the representation apparatus, the captureunit may include an optical and/or electromagnetic and/or acousticsensor for capturing the relative positioning.

For example, the capture unit may have an optical sensor configured as acamera (e.g., a 2D camera and/or an omni-directional camera and/or a 3Dcamera, such as a stereo camera and/or a depth camera and/or atime-of-flight (TOF) camera) that is designed for at least partial(e.g., optical) capture of the first display unit and/or the seconddisplay unit. Further, the capture unit may have an electromagneticsensor that may be configured to locate the first display unit and/orthe second display unit based on electromagnetic waves (e.g., a changeand/or interference of electromagnetic waves). The electromagneticsensor may further be configured as a gyroscopic sensor for capturing aspatial positioning of the capture unit (e.g., if the capture unit isintegrated at least partially into the first display unit or the seconddisplay unit). Further, the capture unit may have an acoustic (e.g.,ultrasound-based) sensor that is configured to emit a defined ultrasonicfield and to capture the relative positioning based on a reflectedportion of the ultrasonic field. In one embodiment, the capture unit maybe configured to provide a signal dependent upon the captured relativepositioning of the first display unit in relation to the representationarea of the second display unit to the processing unit.

The embodiment may enable a precise capture of the relative positioningof the first display unit in relation to the representation area of thesecond display unit.

In a further embodiment of the representation apparatus, the captureunit may be configured to capture the relative positioning based onphysical features of the first display unit and/or the second displayunit and/or based on a graphical marker structure displayed visiblyand/or invisibly on the representation area of the second display unit.

The physical features of the first display unit and/or the seconddisplay unit may include, for example, a contour and/or a shape and/or amaterial property and/or a texture (e.g., a reflectivity and/orabsorptivity). Further, the first display unit and/or the second displayunit may have a physical (e.g., two-dimensional or three-dimensional)marker structure that is fastened in a defined arrangement on the firstdisplay unit and/or second display unit and/or is integrated at leastpartially into the first display unit and/or second display unit (e.g.,on an edge and/or frame of the first display unit and/or the seconddisplay unit). In one embodiment, the capture unit may be configured forcapturing a spatial positioning (e.g., a spatial position and/ororientation and/or posture) of the respective marker structure.

Alternatively or additionally, the second display unit may be configuredto display a graphical marker structure (e.g., for a human user) visiblyand/or non-visibly. The graphical marker structure may include anarrangement of graphical position markers of, for example, a geometricalform and/or a defined pattern that is stationary (e.g., in relation tothe representation area of the second display unit). The capture unitmay be configured to capture the graphical marker structure that isdisplayed in the operating state of the representation apparatus on therepresentation area of the second display unit. The capture unit mayfurther be configured to capture the relative positioning of the firstdisplay unit in relation to the representation area of the seconddisplay unit based on the captured graphical marker structure (e.g.,based on a spatial positioning of the graphical marker structure).

The second display unit may be configured to display the graphicalmarker structure visibly (e.g., at least partially with the graphicalinformation overlaid and/or at least partially integrated into thegraphical information).

Alternatively or additionally, the second display unit may be configuredto display the graphical marker structure non-visibly. Therein, thedisplay of the graphical marker structure may take place in a wavelengthregion outside a light spectrum perceptible to the human user (e.g., inan infrared region). Alternatively or additionally, the second displayunit may be configured to display a graphical marker structuretemporally and/or spatially interlaced with the graphical information.The temporal interlacing may describe a display (e.g., repeated display)of the graphical marker structure within a temporal sequence of thedisplay of the graphical information. Therein, the second display unitmay be configured to display the graphical marker structure so seldomand/or briefly within the sequence of the display of the graphicalinformation that the graphical marker structure is not perceptible tothe human user. The spatially interlaced display of the graphical markerstructure with the graphical information may take place, for example,line by line and/or column by column. Therein, the second display unitmay be configured to specify a spatial resolution of the graphicalmarker structure such that the graphical marker structure in thespatially interlaced display is not perceptible to the human user. Inone embodiment, the second display unit may be configured to display thegraphical marker structure without any influence that is perceptible tothe human user on the graphical information.

The embodiment may enable a particularly robust and simultaneouslyprecise capture of the relative positioning of the first display unit inrelation to the representation area of the second display unit.

In a further embodiment of the representation apparatus, the graphicalrepresentation of the augmented reality may have a graphicalrepresentation of at least one virtual object of the augmented reality.Further, the capture unit may be configured to determine an observationparameter based on the relative positioning. Further, the processingunit may be configured to determine the at least one virtual objectbased on the observation parameter. Alternatively or additionally, theprocessing unit may be configured to adapt a resolution and/or scalingand/or size and/or positioning of the graphical representation of the atleast one virtual object based on the observation parameter.

The observation parameter determined based on the relative positioningmay describe one or more properties of the observation geometry (e.g.,not user-specifically). For example, the observation parameter mayinclude an item of information regarding, for example, a momentaryspatial distance between the first display unit (e.g., the origin of thecoordinate system of the first display unit) and the representation areaof the second display unit (e.g., the reference point, such as themidpoint or the corner point of the representation area). Further, theobservation parameter may include an item of information regarding theangle (e.g., the viewing angle) of the first display unit in relation tothe representation area of the second display unit (e.g., the normalvector in the reference point of the representation area). Further, theobservation parameter may include an item of information regarding thespatial extent (e.g., a fan angle) of the field of view of the firstdisplay unit in relation to the representation area of the seconddisplay unit.

The processing unit may be configured to determine the at least onevirtual object based on the observation parameter. Therein, thedetermination of the at least one virtual object may include, forexample, a filtration of, for example, content and/or graphics and/or areconstruction of the dataset. The filtration of the dataset mayinclude, for example, a selection of elements and/or features and/orinformation of the dataset based on the observation parameter fordetermining the at least one virtual object of the dataset.Alternatively or additionally, the processing unit may be configured toadapt the, for example, spatial and/or temporal resolution and/orscaling and/or the size and/or the positioning of the graphicalrepresentation of the at least one virtual object based on theobservation parameter. For example, the processing unit may beconfigured to reduce the spatial resolution of the graphicalrepresentation of the at least one virtual object with increasingspatial distance between the first display unit and the representationarea of the second display unit. In addition, the processing unit may beconfigured to reduce the graphical representation of the at least onevirtual object with increasing spatial distance between the firstdisplay unit and the representation area of the second display unit, forexample, by scaling. In addition, the processing unit may be configuredto adapt the positioning (e.g., the position and/or orientation) of thegraphical representation of the at least one virtual object on therepresentation area of the second display unit dependent upon theobservation parameter (e.g., the viewing angle). The processing unit mayfurther be configured for adapting further representation parameters ofthe graphical representation of the at least one virtual object based onthe observation parameter (e.g., a script size and/or a line thicknessand/or a color coding and/or a grey scale coding).

The embodiment may enable an immersive and realistic display of thegraphical representation of the augmented reality (e.g., of the at leastone virtual object in the at least partial overlaying with therepresentation area of the second display unit).

In a further embodiment of the representation apparatus, the captureunit may be configured to capture an input and/or identification of auser who looks through the first display unit in an operating state ofthe representation apparatus. Further, the capture unit may beconfigured to determine the observation parameter additionally based onthe input and/or identification of the user (e.g., user-specifically).

The sensor of the capture unit for capturing the relative positioningmay also be configured to capture the input of the user. Alternativelyor additionally, the capture unit may have a further sensor (e.g.,optical and/or electromagnetic and/or acoustic and/or haptic) that isconfigured to capture the input of the user. The capture unit may alsobe configured to capture the input of the user by use of an inputdevice. The input device may include, for example, a pointing device(e.g., a stylus and/or a marker structure) and/or an input unit (e.g., akeyboard) and/or a body part of the user (e.g., a hand and/or a finger)and/or an optical and/or acoustic signal. In one embodiment, the captureunit may be configured for two-dimensional and/or three-dimensionalspatial capture of the input of the user, for example, based on theinput device. For example, the capture unit may be configured to capturethe input of the user point-wise and/or in a time resolved manner (e.g.,as a trajectory and/or a gesture).

The input by the user may have an item of information (e.g., at leastone input parameter for the determination of the at least one virtualobject and/or the adaptation of the resolution and/or scaling and/or thesize and/or positioning of the graphical representation of the at leastone virtual object). For example, the input by the user may have acriterion for the selection of elements and/or features and/orinformation of the dataset for determining the at least one virtualobject.

Further, the capture unit may be configured to capture theidentification of the user. The capture unit may be configured tocapture the identification of the user using the sensor of the captureunit for capturing the relative positioning and/or using the furthersensor. The capture of the identification of the user may take place,for example, using biometric features (e.g., a voice profile and/or afingerprint and/or using the input of the user and/or using anidentification medium, such as a barcode and/or a radio frequencyidentification (RFID) system). The processing unit may be configured toassign a role based on the identification of the user. The differentroles of the user may be classified, for example, based on anobservation interest (e.g., medical observation interest) and/or aninteraction level and/or an expertise of the respective user.

The processing unit may be configured to generate the augmented realityat least partially differently for different identifications (e.g.,roles) of users. Further, the processing unit may be configured toprovide the graphical representation of the augmented reality at leastpartially differently for different identifications (e.g., roles) ofusers. The capture unit may be configured to determine (e.g., to select)the observation parameter additionally based on the identification(e.g., the role) of the user. In one embodiment, the capture unit maydetermine at least one observation parameter regarding the classifiedroles of users via a capture and/or a readout of a computer-readabledata store and/or a reception from a data storage unit (e.g., adatabase). The observation parameter may have, for example, theuser-specific and/or role-specific criterion for the selection ofelements and/or features and/or information of the dataset fordetermining the at least one virtual object. Alternatively oradditionally, the observation parameter may have a, for example,user-specific and/or role-specific instruction regarding the resolutionand/or scaling and/or size and/or positioning of the graphicalrepresentation of the at least one virtual object. Further, the, forexample, user-specific observation parameter may have an item ofinformation regarding a user specific stereo disparity. Therein, theprocessing unit may be configured to adapt the virtual mapping forproviding the graphical representation of the augmented reality (e.g.,for a stereoscopic display of the graphical representation of theaugmented reality based on the observation parameter to theuser-specific stereo disparity).

The embodiment may enable a user-specific and/or context-sensitiveadaptation (e.g., manual adaptation) of the graphical representation ofthe augmented reality (e.g., of the at least one virtual object).

In a further embodiment of the representation apparatus, the dataset mayhave a parameter relating to the graphical information that the seconddisplay unit displays in an operating state of the representationapparatus on the representation area. Therein, the processing unit mayfurther be configured to generate the augmented reality dependent uponthe parameter.

The parameter may have an item of origin information and/or an item ofrepresentation information regarding the graphical information thatdisplays the second display unit in the operating state of therepresentation apparatus. Therein, the origin information may have, forexample, an item of information relating to a source of the graphicalinformation (e.g., a medical imaging device for recording and/or forproviding the graphical information). For example, the parameter mayhave metadata relating to the graphical information. Further, the origininformation may have a recording parameter and/or a reconstructionparameter relating to the graphical information. In addition, the origininformation may have an item of positioning information relating, forexample, to a spatial position and/or orientation and/or posture and/ora spatial recording region of the source (e.g., of the medical imagingdevice) for recording and/or for providing the graphical information.The representation information may further have an item of informationregarding the display of the graphical information (e.g., an imagefrequency and/or a resolution and/or an encoding of image values, suchas a color-coding or a gray-scale coding).

The processing unit may further be configured to generate the augmentedreality dependent upon the parameter. For example, the processing unitmay be configured to determine the at least one virtual object (e.g.,the plurality of virtual objects) dependent upon the parameter. Further,the processing unit may be configured to adapt the virtual arrangementof the at least one virtual object (e.g., the plurality of virtualobjects) in the coordinate system of the first display unit dependentupon the parameter. By this, the positioning of the graphicalrepresentation of the at least one virtual object may be adapted to therepresentation area of the second display unit dependent upon theparameter. For example, the processing unit may be configured toregister the augmented reality (e.g., the at least one virtual object)based on the observation geometry and the parameter with the graphicalinformation. The processing unit may further be configured to adapt thegraphical representation of the augmented reality (e.g., the at leastone virtual object dependent upon the parameter).

By this, an improved immersion of the graphical representation of theaugmented reality in the at least partial overlaying with the graphicalinformation displayed on the representation area of the second displayunit in the operating state may be enabled.

In a further embodiment of the representation apparatus, the dataset mayhave first medical image data. Therein, the processing unit may beconfigured to generate the augmented reality based on the first medicalimage data. In one embodiment, in the operating state of therepresentation apparatus, the second display unit may display agraphical representation of second medical image data on therepresentation area as the graphical information. Therein, the firstmedical image data and the second medical image data may have a mappingand/or a model of an at least partial shared examination region (e.g.,common examination region) of an examination object. Further, in theoperating state of the representation apparatus, the graphicalrepresentation of the augmented reality may be at least partiallyoverlaid with the graphical representation of the second medical imagedata.

The first medical image data and/or the second medical image data may berecorded and/or provided by a medical imaging device or differentmedical imaging devices. The at least one medical imaging device may beconfigured for recording the first medical image data and/or the secondmedical image data, for example, as a medical X-ray device and/or amagnetic resonance (MRT) system and/or a computed tomography (CT) systemand/or a positron emission tomography (PET) system and/or anultrasonography device and/or an endoscope (e.g., a laparoscope and/or abronchoscope and/or catheter). Therein, the first medical image data andthe second medical image data may be the same or different with regardto recording parameters and/or reconstruction. Further, the firstmedical image data and the second medical image data may be registeredto one another.

In one embodiment, the first medical image data may have a twodimensional or three-dimensional spatially resolved (e.g., and timeresolved) mapping of at least a first examination region (e.g., ananatomical region and/or an organ, such as a hollow organ and/or a bonestructure of the examination object). Further, the second medical imagedata may have a two dimensional or three-dimensional spatially resolved(e.g., and time resolved) mapping of at least a second examinationregion (e.g., an anatomical region and/or an organ, such as a holloworgan and/or a bone structure of the examination object). The firstmedical image and/or the second medical image data may have, forexample, a scene (e.g., a surgical video). Therein, the firstexamination region and the second examination region may match at leastpartially. By this, the first medical image data and the second medicalimage data may at least partially map a shared examination region of theexamination object. The first medical image data and/or second medicalimage data may each have a contrasted and/or segmented mapping of theexamination object (e.g., the first examination region and/or the secondexamination region). The examination object may be, for example, a humanand/or animal patient and/or an examination phantom. Further, the firstmedical image data and the second medical image data may map theexamination object (e.g., the shared examination region) at different orat least partially the same recording time points (e.g., preoperativelyand/or intraoperatively). Further, the first medical image data may havea, for example, intraoperative mapping of the medical imaging device(e.g., in an embodiment as an endoscope and/or a catheter) for recordingthe second medical image data, or vice versa.

Alternatively or additionally, the first medical image data and/or thesecond medical image data may have a 2D and/or 3D model (e.g., acenterline model and/or a volume model, such as a volume mesh model) ofthe respective examination region (e.g., of the hollow organ).

The processing unit may be configured to generate the augmented realitybased on the first medical image data. For example, the processing unitmay be configured to determine the at least one virtual object bysegmentation and/or identification of anatomical objects (e.g., a tissueregion and/or an organ, such as a hollow organ and/or medical objects,such as a surgical and/or diagnostic instrument and/or an implant) thatare mapped in the first medical image data. The at least one virtualobject may have, for example, a mapping and/or a virtual representation(e.g., a 2D or 3D model) of the segmented and/or identified anatomicaland/or medical objects.

The second display unit may be configured to display the graphicalrepresentation of the second medical image data as the graphicalinformation in the operating state of the representation apparatus.Further, the processing unit may be configured to provide the graphicalrepresentation of the augmented reality such that the graphicalrepresentation of the augmented reality displayed by the first displayunit in the operating state of the representation apparatus is at leastpartially overlaid with the graphical representation of the secondmedical image data on the representation area of the second displayunit. For example, the processing unit may be configured to provide thegraphical representation of the augmented reality such that the at leastpartially shared region of the examination object is arrangedcongruently in the overlay of the graphical representation of the secondmedical image data with the graphical representation of the augmentedreality. For example, the processing unit may be configured to registerthe graphical representation of the augmented reality with the graphicalrepresentation of the second medical image data based on the relativepositioning (e.g., the observation geometry).

The embodiment may enable an at least partially overlaid display of thegraphical representation of the augmented reality with the graphicalrepresentation of the second medical image data displayed on therepresentation area of the second display unit in the operating state.Therein, the display of the graphical representation of the secondmedical image data (e.g., with regard to image quality and/or a displaydelay) may remain uninfluenced by the display of the graphicalrepresentation of the augmented reality.

In a further embodiment of the representation apparatus, the first imagedata may map and/or model the examination object three-dimensionally.Further, the processing unit may be configured to provide the graphicalrepresentation of the augmented reality having a virtual window relatingto the graphical representation of the second image data.

In one embodiment, the processing unit may be configured to determineand/or identify a region of interest (e.g., two-dimensional region ofinterest) on the representation area of the second display unit (e.g.,in the second medical image data). Therein, the processing unit may beconfigured to determine and/or identify the region of interest, forexample, based on a further input by the user and/or based ongeometrical features (e.g., a contour and/or an image value and/or acontrast value and/or anatomical features, such as a tissue boundaryand/or a vessel wall and/or a marker structure of the second medicalimage data). In one embodiment, the region of interest may delimit atleast one part (e.g., coherent and/or spatially limited) of therepresentation area of the second display unit on which the virtualwindow is to be positioned in the graphical representation of theaugmented reality.

The virtual window may describe a spatially limited and/or coherentregion of the graphical representation of the augmented reality that, inthe at least partial overlaying with the representation area of thesecond display unit, has a virtual transparency in relation to thegraphical representation of the second medical image data. In oneembodiment, the processing unit may be configured to arrange thegraphical representation of the augmented reality (e.g., the graphicalrepresentation of the at least one virtual object) on the region ofinterest on the representation area of the second display unit. Theprocessing unit may be configured to provide the virtual windowtwo-dimensionally or three-dimensionally spatially resolved. The virtualwindow may further have a, for example, two-dimensionally orthree-dimensionally spatially resolved region of the first medical imagedata. In one embodiment, the processing unit may be configured toprovide the graphical representation of the augmented reality such thatthe at least partially shared region of the examination object (e.g.,shared geometrical and/or anatomical features) in the overlaying of thegraphical representation of the second medical image data with thegraphical representation of the augmented reality (e.g., the virtualwindow) is arranged congruently. The processing unit may also beconfigured to adapt the virtual window (e.g., the region of the firstmedical image data dependent upon the momentary observation geometry).Further, the processing unit may be configured to adapt the virtualwindow dependent upon the observation parameter and/or the parameterrelating to the graphical information (e.g., the second medical imagedata). The adaptation of the virtual window dependent upon theobservation geometry may include the virtual mapping of the augmentedreality onto the representation area of the second display unit (e.g.,the region of interest) having a depth dimension in relation to therepresentation area. Therein, the virtual mapping of the augmentedreality (e.g., the at least one virtual object) onto the representationarea (e.g., on a virtual projection mapping and/or ray-tracing and/or arendering in accordance with a mapping matrix, such as a mapping matrixand/or camera matrix of the medical imaging device for recording and/orfor providing the first medical image data) may be based along theobservation geometry. Further, the depth dimension may extendsubstantially perpendicularly to the representation area of the seconddisplay unit. For example, the region of interest on the representationarea of the second display unit may delimit the spatial region of thetransparency that is provided by the overlaying with the graphicalrepresentation of the augmented reality (e.g., the virtual window).

The embodiment may enable an immersive observation of the first medicalimage data in the graphical representation of the augmented reality viathe virtual window in the representation area of the second displayunit.

In a further embodiment of the representation apparatus, the sharedregion of the examination object may have a bordered hollow space.Therein, the virtual window may be delimited by the bordering of thehollow space cut from a virtual plane in the virtual mapping of theaugmented reality onto the representation area of the second displayunit.

The bordered hollow space may denote a spatially delimited space (e.g.,a fluid-filled hollow space) of the examination object, spatiallydelimited, for example, by a tissue and/or a structure and/or a tissueboundary (e.g., a hollow organ). The hollow organ may include, forexample, a vessel portion (e.g., an artery or a vein and/or a lungand/or a gut and/or a heart).

In one embodiment, the processing unit may be configured to determineand/or identify the region of interest on the representation area of thesecond display unit (e.g., in the second medical image data) based onthe border of the hollow space. In the virtual mapping of the augmentedreality onto the representation area of the second display unit, theborder of the hollow space may be cut by a virtual plane that is, forexample, parallel or tilted to the representation area of the seconddisplay unit. Therein, the cut border of the hollow space may have a,for example, closed or interrupted contour. The processing unit may beconfigured to determine and/or identify the contour of the cut border ofthe hollow space (e.g., in the virtual mapping of the augmentedreality), for example, based on image values and/or contrast values. Inone embodiment, the processing unit may be configured to determineand/or identify the region of interest as a, for example, coherent areathat is bordered by the contour of the cut border. For example, thecontour of the cut hollow space may delimit the spatial region of thetransparency that is provided via the overlaying with the graphicalrepresentation of the augmented reality (e.g., the virtual window). In atwo-dimensional design of the virtual window, the processing unit may beconfigured to determine the virtual window in a spatially limitedmanner, according to the region of interest (e.g., via the contour ofthe cut border of the hollow space). In a three-dimensional design ofthe virtual window, the processing unit may be configured to delimit thevirtual window additionally along the depth dimension of therepresentation area via the border of the hollow space.

By this, a representation of the bordered hollow space that is close toreality, and its observation via the virtual window, may be enabled.

In a further embodiment of the representation apparatus, the processingunit may be configured to provide the graphical representation of theaugmented reality having a virtual continuation of at least one objectmapped in the second image data. Further, the first display unit may beconfigured to display the virtual continuation outside and/or adjoiningthe representation area of the second display unit.

In one embodiment, the processing unit may be configured to identify theat least one object (e.g., anatomical and/or medical and/or geometricalobject) in the second medical image data, for example, based on imagevalues and/or contrast values and/or a marker structure and/or ananatomy atlas. Therein, the identification of the at least one object inthe second medical image data may include, for example, a segmentation.Further, the processing unit may be configured to identify an object inthe first medical image data that corresponds to the at least oneobject, for example, based on the registration between the first medicalimage data and the second medical image data and/or based on imagevalues and/or contrast values and/or a marker structure and/or ananatomy atlas. Therein, the identification of the at least onecorresponding object in the second medical image data may include, forexample, a segmentation. In one embodiment, the processing unit mayfurther be configured to determine the virtual continuation having amapping and/or a virtual representation (e.g., a model) of thecorresponding object based on the first medical image data. Further, theprocessing unit may be configured to map the augmented reality virtuallyonto the representation area of the second display unit such that thevirtual continuation in the operating state of the representationapparatus is displayed outside and/or adjoining the representation areaof the second display unit. Therein, the virtual continuation may bearranged in a plane parallel to the representation area of the seconddisplay unit or having a predetermined angle in relation to therepresentation area.

By this, a display of the graphical representation of the augmentedreality that is registered to the representation area of the seconddisplay unit and is not spatially delimited by the representation areamay be enabled.

The present embodiments relate, in a second aspect, to a systemincluding a representation apparatus (e.g., a plurality ofrepresentation apparatuses), a second display unit, and a medicalimaging device. Therein, the medical imaging device is configured torecord and/or provide the first medical image data and/or the secondmedical image data. Further, the second display unit is configured todisplay a graphical representation of the second medical image data onthe representation area.

The advantages of the system substantially correspond to the advantagesof the representation apparatus for displaying a graphicalrepresentation of an augmented reality. Features, advantages, oralternative embodiments mentioned herein may also be transferred to theother subject matter and vice versa.

In a further embodiment of the system, the medical imaging device may beconfigured to be arranged at least partially in an examination region ofan examination object.

The medical imaging device may be configured, for example, as anendoscope (e.g., a laparoscope and/or a bronchoscope and/or a catheter).Therein, a distal portion of the medical imaging device may be arranged,in an operating state of the system, in the examination object (e.g., ina hollow organ of the examination object). In addition, the distalportion of the medical imaging device may be configured for recordingthe first medical image data and/or the second medical image data.

The present embodiments relate, in a third aspect, to a method forproviding a graphical representation of an augmented reality. Therelative positioning of a first display unit in relation to arepresentation area of a second display unit is captured by a captureunit. Further, graphical information is displayed on the representationarea of the second display unit. Further, an observation geometrybetween the first display unit and the representation area of the seconddisplay unit is determined based on the relative positioning. Inaddition, a dataset is received. Further, the augmented reality isgenerated based on the dataset. Further, the graphical representation ofthe augmented reality is provided via a virtual mapping of the augmentedreality onto the representation area of the second display unit alongthe observation geometry. Therein, the provision includes a display ofthe graphical representation of the augmented reality by the firstdisplay unit in at least partial overlaying with the graphicalinformation on the representation area of the second display unit.

The advantages of the method substantially correspond to the advantagesof the representation apparatus for displaying a graphicalrepresentation of an augmented reality. Features, advantages, oralternative embodiments mentioned herein may also be transferred to theother subject matter and vice versa. The apparatuses according to thepresent embodiments (e.g., the representation apparatus and/or thesystem) may be configured to carry out an embodiment of the method.

BRIEF DESCRIPTION OF THE DRAWINGS

Exemplary embodiments of the invention are illustrated in the drawingsand are described in greater detail below. In the different figures, thesame reference signs are used for the same features. In the figures:

FIGS. 1 to 3 are schematic representations of different embodiments of arepresentation apparatus for displaying a graphical representation of anaugmented reality;

FIG. 4 is a schematic representation of an example of a virtual mappingof the augmented reality having a virtual window;

FIG. 5 is a schematic representation of an embodiment of a system; and

FIG. 6 is a schematic representation of one embodiment of a method forproviding a graphical representation of an augmented reality.

DETAILED DESCRIPTION

FIG. 1 shows an embodiment of a representation apparatus for displayinga graphical representation of an augmented reality, illustratedschematically. The representation apparatus may have a capture unit RU,a first display unit VIS.1, and a processing unit 22. Therein, the firstdisplay unit VIS.1 may be configured to be at least partiallytransparent. Further, the capture unit RU may be configured to capture arelative positioning of the first display unit VIS.1 in relation to arepresentation area VIS.S of a second display unit VIS.2. The captureunit RU may have, for example, an optical and/or electromagnetic and/oracoustic sensor for capturing the relative positioning. Further, therepresentation area VIS.S of the second display unit VIS.2 may beconfigured to display graphical information GI. The second display unitVIS.2 may include, for example, a screen and/or a monitor and/or aprojector and/or a projection area. Therein, the representation areaVIS.S of the second display unit VIS.2 may include the projection areaand/or a display layer of the screen and/or the monitor. For example,the representation area VIS.S of the second display unit VIS.2 mayinclude a limited area on which the graphical information GI may bedisplayed. Therein, the representation area VIS.S of the second displayunit VIS.2 may extend, at least partially (e.g., completely) flat (e.g.,planar or curved). The graphical information GI may include image data(e.g., medical image data) and/or text information (e.g., operatingparameters and/or object parameters of an object (medical object) and/ormetadata regarding the medical image data.

In addition, the processing unit 22 is configured to determine anobservation geometry between the first display unit VIS.1 and therepresentation area VIS.S of the second display unit VIS.2 based on therelative positioning. The processing unit 22 may be further configuredto receive a dataset. In one embodiment, the processing unit 22 may becommunicatively coupled to the first display unit VIS.1 and the captureunit RU, for example, by a signal 22.S. Further, the processing unit 22may be configured to generate the augmented reality based on thedataset. Further, the processing unit 22 may be configured to provide agraphical representation G.AR of the augmented reality via a virtualmapping of the augmented reality onto the representation area VIS.S ofthe second display unit VIS.2 along the observation geometry.

The first display unit VIS.1 may be configured to display the graphicalrepresentation G.AR of the augmented reality in at least partialoverlaying with the representation area VIS.S of the second display unitVIS.2 (e.g., stereoscopically). For this purpose, the first display unitVIS.1 may have, for example, a screen and/or a monitor and/or aprojector and/or a projection area. In one embodiment, the first displayunit VIS.1 may be configured as goggles (e.g., data goggles and/or ahelmet, such as a data helmet). Further, the first display unit VIS.1may be configured to be portable (e.g., wearable by a user U within afield of view of the user U).

Therein, the capture unit RU may be integrated at least partially intothe first display unit VIS.1. In addition, the capture unit RU may havea defined (e.g., positionally fixed) arrangement in relation to thefirst display unit VIS.1. In one embodiment, the capture unit RU may beintegrated at least partially into the first display unit VIS.1 suchthat the second display unit VIS.2 is arranged, in the operating stateof the representation apparatus, in a capture region of the capture unitRU.

In one embodiment, the processing unit 22 may be further configured,given an altered relative positioning, to determine repeatedly theobservation geometry between the first display unit VIS.1 and therepresentation area VIS.S of the second display unit VIS.2. In addition,the processing unit 22 may be configured, with a changed relativepositioning, to provide the graphical representation G.AR of theaugmented reality based on the most recently determined observationgeometry.

The graphical representation G.AR of the augmented reality may have agraphical representation of at least one virtual object of the augmentedreality. Therein, the capture unit RU may be configured to determine anobservation parameter based on the relative positioning. Further, theprocessing unit 22 may be configured to determine the at least onevirtual object based on the observation parameter. Alternatively oradditionally, the processing unit 22 may be configured to adapt aresolution and/or scaling and/or size and/or positioning of thegraphical representation of the at least one virtual object based on theobservation parameter.

The capture unit RU may further be configured to capture an input and/oran identification of the user U who looks through the first display unitVIS.1 in an operating state of the representation apparatus. Inaddition, the capture unit RU may be configured to determine theobservation parameter additionally based on the input and/oridentification of the user U.

In one embodiment, the dataset may have a parameter relating to thegraphical information GI that the second display unit VIS.2 displays inan operating state of the representation apparatus on the representationarea VIS.S. Further, the processing unit 22 may be configured togenerate the augmented reality dependent upon the parameter.

FIG. 2 shows a schematic representation of a further embodiment of therepresentation apparatus. Therein, the capture unit RU may be arranged,spaced from the first display unit VIS.1 and the second display unitVIS.2 (e.g., positionally fixed in relation to a room in which the firstdisplay unit VIS.1 and the second display unit VIS.2 are arranged in anoperating state of the representation apparatus). In one embodiment, thecapture unit RU may be arranged (e.g., positioned), such that the firstdisplay unit VIS.1 and the second display unit VIS.2 are arranged, inthe operating state of the representation apparatus, in a capture regionof the capture unit RU. In addition, the capture unit RU may beconfigured to directly capture the relative positioning based onphysical features of the first display unit VIS.1 and the second displayunit VIS.2.

FIG. 3 shows schematically a further embodiment of the representationapparatus. Therein, the capture unit RU may be configured to capture therelative positioning based on a graphic marker structure PM displayedvisibly and/or non-visibly on the representation area VIS.S of thesecond display unit VIS.2. The graphical marker structure PM may includea two-dimensional arrangement of graphical position markers of, forexample, a geometrical form and/or a defined pattern that is stationary,for example, in relation to the representation area VIS.S of the seconddisplay unit VIS.2. Further, the capture unit RU may be configured tocapture the relative positioning of the first display unit VIS.1 inrelation to the representation area VIS.S of the second display unitVIS.2 based on the captured graphical marker structure (e.g., based on aspatial positioning of the graphical marker structure PM).

FIG. 4 shows a schematic representation of a virtual mapping of theaugmented reality AR having a virtual window WW. Therein, the datasetmay have first medical image data. In addition, the processing unit 22may be further configured to generate the augmented reality AR based onthe first medical image data. Further, in the operating state of therepresentation apparatus, the second display unit VIS.2 may display agraphical representation of second medical image data on therepresentation area VIS.S. Therein, the first medical image data and thesecond medical image data may form a mapping and/or a model of an atleast partial shared examination region of an examination object. In oneembodiment, the first medical image data may map the examination objectthree-dimensionally. Further, the graphical representation G.AR of theaugmented reality AR may be at least partially overlaid with thegraphical representation of the second medical image data. Further, theprocessing unit 22 may be configured to provide the graphicalrepresentation G.AR of the augmented reality AR having a virtual windowWW in relation to the graphical representation of the second image data.

In one embodiment, the shared region of the examination object may havea bordered hollow space HO (e.g., a hollow organ). The hollow organ mayinclude, for example, a vessel portion (e.g., an artery or a vein and/ora lung and/or a gut structure and/or a heart). Therein, the virtualwindow WW may be delimited by the border VW of the hollow space HO cutfrom a virtual plane VP in the virtual mapping of the augmented realityAR onto the representation area VIS.S of the second display unit VIS.2.In the virtual mapping of the augmented reality AR onto therepresentation area VIS.S of the second display unit VIS.2, the borderVW of the hollow space HO may be cut by the virtual plane VP, which isparallel, for example, to the representation area VIS.S of the seconddisplay unit VIS.2. Therein, the cut border VW of the hollow space HOmay have a, for example, closed or interrupted contour WW.C. Forexample, the contour WW.C of the cut border VW of the hollow space HOmay delimit the spatial region of the transparency that is provided bythe overlaying, with the graphical representation G.AR, of the augmentedreality AR (e.g., the virtual window WW). In one embodiment, theprocessing unit 22 may be configured to determine the virtual window WWin a spatially limited manner, according to the border VW of the hollowspace HO cut by the virtual plane VP (e.g., according to the contourWW.C). In a three-dimensional design of the virtual window WW, theprocessing unit 22 may be configured to delimit the virtual window WWadditionally along the depth dimension of the representation area VIS.Svia the border VW of the hollow space HO. For illustration, in FIG. 4 ,two exemplary observation positions P.1 and P.2 of the user U lookingthrough the first display unit are shown. Therein, the display of thegraphical representation of the second image data overlaid with thegraphical representation G.AR of the augmented reality AR may beobserved by the user U on the representation area VIS.S of the seconddisplay unit VIS.2 dependent upon the observation position P.1 or P.2(e.g., momentary observation position P.1 or P.2) with a differentviewing direction VD.1 or VD.2. Therein, the limit rays VD.1B to theviewing direction VD.1 and the limit rays VD2.B to the viewing directionVD.2 illustrate the limitation (e.g., visual limitation) of the virtualwindow WW along the depth dimension of the representation area VIS.S bythe border VW of the hollow space HO. Therein, the virtual mapping ofthe augmented reality AR onto the representation area VIS.S of thesecond display unit VIS.2 for providing the graphical representationG.AR of the augmented reality AR may be based, for example, on a virtualprojection mapping and/or ray-tracing and/or a rendering in accordancewith a mapping matrix (e.g., a mapping matrix and/or camera matrix ofthe medical imaging device for recording and/or for providing the firstmedical image data), along the observation geometry (e.g., dependentupon the observation position P.1 or P.2 and/or the viewing directionVD.1 or VD.2 of the user U).

FIG. 5 shows schematically one embodiment of a system. The system mayhave the representation apparatus, the second display unit VIS.2, and amedical imaging device MD. Therein, the medical imaging device MD may beconfigured to be arranged at least partially in an examination region RE(e.g., a hollow organ of an examination object 31 arranged on a patientpositioning apparatus 32). In one embodiment, the medical imaging deviceMD may be configured to record and/or provide the first medical imagedata and/or the second medical image data. Further, the second displayunit VIS.2 may be configured to display the graphical representation ofthe second medical image data on the representation area VIS.S.

The processing unit 22 may be configured to provide the graphicalrepresentation G.AR of the augmented reality AR having a virtualcontinuation G.VG of at least one object mapped in the second image data(e.g., an anatomical and/or medical and/or geometric object G.OB). FIG.5 shows, by way of example, a hollow organ as the at least one objectGI.OB mapped in the second image data. Therein, the first display unitVIS.1 may be configured to display the virtual continuation G.VF outsideand/or adjoining the representation area VIS.S of the second displayunit VIS.2. Therein, the virtual continuation G.VF may have an item ofgraphical path planning information G.PP for a medical object and/or forthe medical imaging device MD in the hollow organ.

FIG. 6 shows schematically an embodiment of a method for providingPROV-G.AR a graphical representation G.AR of an augmented reality AR.Therein, the relative positioning POS of the first display unit VIS.1 inrelation to the representation area VIS.S of the second display unitVIS.2 may be captured by the capture unit RU. Further, the graphicalinformation GI may be displayed VISU-GI on the representation area VIS.Sof the second display unit VIS.2. Further, the observation geometry GEObetween the first display unit VIS.1 and the representation area VIS.Sof the second display unit VIS.2 may be determined DET-GEO based on therelative positioning POS. In addition, the dataset DS may be receivedREC-DS. Further, the augmented reality AR may be generated GEN-AR basedon the dataset DS. Further, the graphical representation of theaugmented reality G.AR may be provided PROV-G.AR via the virtual mappingof the augmented reality AR onto the representation area VIS.S of thesecond display unit VIS.2 along the observation geometry GEO. Therein,the provision PROV-G.AR may include a display VISU-G.AR of the graphicalrepresentation of the augmented reality G.AR using the first displayunit VIS.1 in at least partial overlaying with the graphical informationGI on the representation area VIS.S of the second display unit VIS.2.

The schematic representations contained in the drawings described do notshow any scale or size relationship.

The methods and apparatuses described above in detail merely involveexemplary embodiments that may be modified by a person skilled in theart in a wide variety of ways without departing from the scope of theinvention. Further, the use of the indefinite article “a” or “an” doesnot preclude the possibility that the relevant features may also bepresent plurally. Similarly, the expressions “unit” and “element” do notpreclude the components in question consisting of a plurality ofcooperating subcomponents that may possibly also be spatiallydistributed.

The advantage of the present embodiments is the method and device enableeffective detection of tumor region in the medical image. Further, thepresent embodiments enable identification of right feeder vessels forembolization. Therefore, healthy tissues associated with the patient arenot damaged due to cancer therapy. Additionally, the present embodimentsreduce the need for manual identification of tumor region in the medicalimages. The present embodiments further enable targeted cancer therapybased on which the tumor blood vessel network may be embolizedaccurately. Further, the present embodiments enable multiple tumors tobe embolized in one medical procedure.

The foregoing examples have been provided merely for the purpose ofexplanation and are in no way to be construed as limiting of the presente disclosed herein. While the invention has been described withreference to various embodiments, it is understood that the words, whichhave been used herein, are words of description and illustration, ratherthan words of limitation. Further, although the invention has beendescribed herein with reference to particular means, materials, andembodiments, the invention is not intended to be limited to theparticulars disclosed herein; rather, the invention extends to allfunctionally equivalent structures, methods, and uses, such as arewithin the scope of the appended claims. Those skilled in the art,having the benefit of the teachings of this specification, may effectnumerous modifications thereto, and changes may be made withoutdeparting from the scope and spirit of the invention in its aspects.

1. A representation apparatus for displaying a graphical representationof an augmented reality, the representation apparatus comprising: acapture unit; a first display unit; and a processing unit, wherein thefirst display unit is configured to be at least partially transparent,wherein the capture unit is configured to capture a relative positioningof the first display unit in relation to a representation area of asecond display unit, wherein the representation area of the seconddisplay unit is configured to display graphical information, wherein theprocessing unit is configured to: determine an observation geometrybetween the first display unit and the representation area of the seconddisplay unit based on the relative positioning; receive a dataset;generate the augmented reality based on the dataset; and provide thegraphical representation of the augmented reality via a virtual mappingof the augmented reality onto the representation area of the seconddisplay unit along the observation geometry, and wherein the firstdisplay unit is configured to display the graphical representation ofthe augmented reality in at least partial overlaying with therepresentation area of the second display unit.
 2. The representationapparatus of claim 1, wherein the processing unit is further configured,in the event of a changed relative positioning, to: determine theobservation geometry between the first display unit and therepresentation area of the second display unit repeatedly; and providethe graphical representation of the augmented reality based on a mostrecently determined observation geometry.
 3. The representationapparatus of claim 1, wherein the capture unit is arranged spaced fromthe first display unit and the second display unit or is integrated atleast partially into the first display unit or the second display unit.4. The representation apparatus of claim 1, wherein the capture unit hasan optical sensor, an electromagnetic sensor, an acoustic sensor, or anycombination thereof for capturing the relative positioning.
 5. Therepresentation apparatus of claim 1, wherein the capture unit isconfigured to capture the relative positioning based on physicalfeatures of the first display unit, the second display unit, or thefirst display unit and the second display unit, based on a graphicmarker structure displayed visibly, non-visibly, or visibly andnon-visibly on the representation area of the second display unit, or acombination thereof.
 6. The representation apparatus of claim 1, whereinthe graphical representation of the augmented reality has a graphicalrepresentation of at least one virtual object of the augmented reality,wherein the capture unit is configured to determine an observationparameter based on the relative positioning, wherein the processing unitis configured to: determine the at least one virtual object based on theobservation parameter; adapt a resolution, scaling, a size, positioning,or any combination thereof of the graphical representation of the atleast one virtual object based on the observation parameter; or acombination thereof.
 7. The representation apparatus of claim 6, whereinthe capture unit is further configured to: capture an input, anidentification, or the input and the identification of a user who looksthrough the first display unit in an operating state of therepresentation apparatus; and determine the observation parameter alsobased on the input, the identification, or the input and theidentification of the user.
 8. The representation apparatus of claim 1,wherein the dataset has a parameter relating to the graphicalinformation, which the second display unit displays on therepresentation area in an operating state of the representationapparatus, and wherein the processing unit is further configured togenerate the augmented reality dependent upon the parameter.
 9. Therepresentation apparatus of claim 1, wherein the dataset includes firstmedical image data, wherein the processing unit is configured togenerate the augmented reality based on the first medical image data,wherein in the operating state of the representation apparatus: thesecond display unit is configured to display a graphical representationof second medical image data as the graphical information on therepresentation area, wherein, the first medical image data and thesecond medical image data have a mapping, a model, or the mapping andthe model of an at least partially common examination region of anexamination object; and the graphical representation of the augmentedreality is at least partially overlaid with the graphical representationof the second medical image data.
 10. The representation apparatus ofclaim 9, wherein the first image data maps, models, or maps and modelsthe examination object three-dimensionally, and wherein the processingunit is configured to provide the graphical representation of theaugmented reality having a virtual window in relation to the graphicalrepresentation of the second image data.
 11. The representationapparatus of claim 10, wherein the common examination region of theexamination object has a bordered hollow space, and wherein the virtualwindow is delimited by the border of the hollow space cut from a virtualplane in the virtual mapping of the augmented reality onto therepresentation area of the second display unit.
 12. The representationapparatus of claim 9, wherein the processing unit is configured toprovide the graphical representation of the augmented reality having avirtual continuation of at least one object mapped into the second imagedata, and wherein the first display unit is configured to display thevirtual continuation outside, adjoining, or outside and adjoining therepresentation area of the second display unit.
 13. A system comprising:a representation apparatus for displaying a graphical representation ofan augmented reality, the representation apparatus comprising: a captureunit; a first display unit; and a processing unit, wherein the firstdisplay unit is configured to be at least partially transparent, whereinthe capture unit is configured to capture a relative positioning of thefirst display unit in relation to a representation area of a seconddisplay unit, wherein the representation area of the second display unitis configured to display graphical information, wherein the processingunit is configured to determine an observation geometry between thefirst display unit and the representation area of the second displayunit based on the relative positioning, receive a dataset, generate theaugmented reality based on the dataset, and provide the graphicalrepresentation of the augmented reality via a virtual mapping of theaugmented reality onto the representation area of the second displayunit along the observation geometry, wherein the first display unit isconfigured to display the graphical representation of the augmentedreality in at least partial overlaying with the representation area ofthe second display unit, wherein the dataset includes first medicalimage data, wherein the processing unit is configured to generate theaugmented reality based on the first medical image data, and wherein inthe operating state of the representation apparatus, the second displayunit is configured to display a graphical representation of secondmedical image data as the graphical information on the representationarea, the first medical image data and the second medical image datahaving a mapping, a model, or the mapping and the model of an at leastpartially common examination region of an examination object, and thegraphical representation of the augmented reality is at least partiallyoverlaid with the graphical representation of the second medical imagedata; the second display unit; and a medical imaging device, wherein themedical imaging device is configured to record, provide, or record andprovide the first medical image data, the second medical image data, orthe first medical image data and the second medical image data, andwherein the second display unit is configured to display a graphicalrepresentation of the second medical image data on the representationarea.
 14. The system of claim 13, wherein the medical imaging device isarrangeable at least partially in an examination region of anexamination object.
 15. A method for providing a graphicalrepresentation of an augmented reality, the method comprising:capturing, by a capture unit, a relative positioning of a first displayunit in relation to a representation area of a second display unit;displaying graphical information on the representation area of thesecond display unit; determining an observation geometry between thefirst display unit and the representation area of the second displayunit based on the relative positioning; receiving a dataset; generatingthe augmented reality based on the dataset; and providing the graphicalrepresentation of the augmented reality via a virtual mapping of theaugmented reality onto the representation area of the second displayunit along the observation geometry, the providing comprising displayingthe graphical representation of the augmented reality by the firstdisplay unit in at least partial overlaying with the graphicalinformation on the representation area of the second display unit. 16.The method of claim 15, wherein in the event of a changed relativepositioning, the method further comprises: determining the observationgeometry between the first display unit and the representation area ofthe second display unit repeatedly; and providing the graphicalrepresentation of the augmented reality based on a most recentlydetermined observation geometry.
 17. The method of claim 15, whereincapturing the relative positioning comprises capturing, by the captureunit, the relative positioning based on physical features of the firstdisplay unit, the second display unit, or the first display unit and thesecond display unit, based on a graphic marker structure displayedvisibly, non-visibly, or visibly and non-visibly on the representationarea of the second display unit, or a combination thereof.
 18. Themethod of claim 15, wherein the graphical representation of theaugmented reality has a graphical representation of at least one virtualobject of the augmented reality, wherein the method further comprises:determining, by the capture unit, an observation parameter based on therelative positioning; and determining the at least one virtual objectbased on the observation parameter, adapting a resolution, scaling, asize, positioning, or any combination thereof of the graphicalrepresentation of the at least one virtual object based on theobservation parameter, or a combination thereof.